Storage systems are used to save data that is used to process a computer. In recent years, multi-node storage systems (dispersion storage systems) have been used that dispersion-manage data using a plurality of storage nodes in order to save a large quantity of data and to enhance reliability.
The multi-node storage system includes a plurality of storage nodes and a storage apparatus connected to each storage node. The plurality of storage nodes are connected via a network. In response to an instruction transmitted through the network, each storage node manages a storage apparatus that is provided within the storage node or connected to the outside and that stores data. In general, in the multi-node storage system, a virtual storage apparatus is defined. In addition, in the multi-node storage system, a redundant configuration is typically provided. In the redundant configuration, redundancy is achieved such that the same data is stored in each of a plurality of logic devices in order to improve, for example, the safety of data. As an example, by storing the same data in two logic devices, the data is duplicated. In this case, the redundancy level is two.
Such a multi-node storage system allows a user to use a storage apparatus connected to a plurality of storage nodes in the same manner as local storage.
In such a multi-node storage system, when a fault occurs in a certain storage apparatus, this storage apparatus becomes inaccessible, thereby breaking the duplicated state of some pieces of data. In this case, to recover the data-duplicated state, a recovery process is performed. In the recovery process, a new portion of a logic device is assigned to data for which the duplicated state has been broken, i.e., data to be recovered. Data is then copied from the existing portion of the logic device assigned to the data to be recovered to the new portion of the logic device. In this way, the data-duplicated state is recovered.
In the multi-node storage system, it is necessary to diagnose whether or not a storage apparatus of the system is being operated normally. For the diagnosis, an existence confirmation is typically performed to diagnose whether the storage apparatus is being operated. The diagnosis via the existence confirmation may be finished in a relatively short time, and, in general, the processing load is not heavy. Accordingly, the diagnosis via the existence confirmation does not greatly affect normal processing by the system.
It is necessary to diagnose, for example, whether data at each storage node having a redundant configuration is adequately holding the redundant configuration. As an example, under a condition in which a redundant configuration is achieved between storage apparatuses, when a fault in one region is found, restoration can be achieved via a recovery process. However, when a fault occurs simultaneously in two regions associated with each other by a redundant configuration, data cannot be restored. As a result, it is necessary to not only perform the existence confirmation for the storage apparatuses but also to diagnose whether or not a fault has occurred for all regions of the storage apparatuses. This process is referred to as a patrol process. The diagnosis method includes, for example, a method wherein data is read to investigate whether an error will occur, a method wherein data that was read is written and is then read again to investigate whether the content is the same, and a method wherein data is read between nodes to confirm whether the content is the same.
In such a multi-node storage system, the performance of the entire system is affected by how data is located among storage apparatuses. For example, access concentration on a particular storage apparatus leads to performance degradation such as a delay in data access. Accordingly, a technology is known wherein, in order to prevent the performance of data access from being degraded, a process is performed of relocating data to achieve proper data locating in accordance with a resource amount of or load on each storage node.
In a multi-node storage system including a plurality of storage nodes and in which data is multiplexed and stored, when a fault occurs in a storage node, a recovery process is performed to recover the redundancy level. As an example, in a multi-node storage system configured by RAID 5, when one storage node fails, a recovery process is performed in which data is copied to a storage apparatus of a new storage node from another normal storage node that stores the same data as data stored in the storage apparatus of the storage node that has failed. In this way, the system recovers the redundancy level. However, during the recovery process, normal processes and the process of recovering the redundancy level are performed in parallel with each other, thereby decreasing, for example, an observable processing speed of the system, such as the speed of a process under a user request.
In the designing stage of a multi-node storage system, it is difficult to predict loads on the entire system that will be applied during a recovery process. Accordingly, it is difficult to design a multi-node storage system such that the system performance is not decreased even when a fault occurs, and observable system performances achieved during a normal operation are thus different from those during the occurrence of a failure.
In view of these factors, there is a problem of difficulty in estimating a time required for the multi-node storage system to perform a certain processing series.
Therefore, a multi-node storage system is required wherein an observable system processing speed achieved during a normal period during which a relocating process and a patrol process are performed in addition to a normal process under an instruction input by a user is the same as an observable system processing speed achieved during a fault occurrence period during which a recovery process is performed in addition to a normal process under an instruction input by the user.    Patent document 1: Japanese Laid-open Patent Publication No. 2009-151519    Patent document 2: Japanese Laid-open Patent Publication No. 05-158625    Patent document 3: Japanese Laid-open Patent Publication No. 2005-050007